Method and device for image processing and a night vision system for motor vehicles

ABSTRACT

A method and a device for image processing and for a night vision system for motor vehicles, in which a standard camera that is sensitive in the near infrared delivers images which are displayed on a display arrangement having a lower resolution than the camera. In addition, the image processing method improves the raw images of the sensor using image-sharpening methods and/or contrast-enhancing methods, so that display on a display arrangement is made possible for viewing by a viewer.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of, and incorporates herein byreference in its entirety, U.S. patent application Ser. No. 10/477,843,which was filed on Nov. 14, 2003 now U.S. Pat. No. 7,474,798 and whichwas a National Stage Application of PCT International Application No.PCT/DE02/04318, filed Nov. 25, 2002.

FIELD OF THE INVENTION

The present invention relates to a method and a device for imageprocessing as well as a night vision system for motor vehicles.

BACKGROUND INFORMATION

One application of video technique in the motor vehicle field is that ofnight vision systems. In this context, heat-sensitive cameras as well asa projection unit may be used, which project the visible images recordedby the camera onto the windshield (so-called head-up displays). U.S.Pat. No. 5,414,439 is an example of such a night vision system. Theexpenditure for such a night vision system may be considerable,particularly because of the high cost of these components.

SUMMARY OF THE INVENTION

By using a camera (for example, a “standard” camera, such as, forexample, an OCD camera or a CMOS camera) that is sensitive to the nearinfrared, particularly in connection with a cost-effective displayarrangement (such as, for example, LCD displays) the expenditure forsuch a night vision system may be considerably reduced.

Furthermore, by sharpening the image of the optical sensor (camera,sensor array) and or by improving the gain in the contrast of the image,the quality of the images recorded in the near infrared may beconsiderably improved, so that it may be used as a night vision systemfor motor vehicles.

The sharpening of the image may be achieved by an image processingmodule, which sharpens the unsharp images created by the reflection ofthe incident infrared light at the rear wall of the housing of theoptical sensor as a result of the transparency to infrared light of thesilicon of the sensor chip. These methods, which may be available inprinciple, enhance the edges of the image, which appear blurred byunsharp imaging. Thereby an image is reconstructed which appears sharpto a viewer. The use of so-called inverse filtering may be particularlyadvantageous.

To enhance the contrast, an adaptive imaging function, which may be anonlinear characteristic function, may be used to image the gray-scalevalues present in the image in the value range of the gray-scale valuesof the display. This allows a sensor having a higher number ofgray-scale values to be used in connection with a display which is onlyable to show fewer gray-scale values. It may be of particular advantagethat the image created, that has its contrast enhanced, displays to theviewer all the essential details, since frequent gray-scale valueregions are shown at high resolution and rarer gray-scale value regionsare shown at low resolution.

Depending on the application example, image sharpening and contrastenhancement may be used jointly, or one procedure may be used by itself.

It may be particularly expedient, when using both modules, to carry outor perform the image sharpening first, and then the contrastenhancement, since the frequency of certain gray-scale values is changedby the image sharpening. This may yield a reliable reconstruction of theoriginally unsharp image that is relatively poor in contrast.

These measures for image processing (image sharpening, contrastenhancement) are also used apart from motor vehicle applications, bothin the infrared and the visible range, when unsharp images have to beprocessed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an image processing system, particularly a night visionsystem for motor vehicles, in which image sharpening and contrastenhancement modules are used.

FIG. 2 shows an exemplary embodiment, which is used for the formation ofthe imaging function for the contrast enhancement.

FIG. 3 shows an exemplary embodiment, which is used for the formation ofthe imaging function for the contrast enhancement.

FIG. 4 shows a flow diagram of an example implementation of thedescribed procedure as a computer program.

DETAILED DESCRIPTION

FIG. 1 shows an arrangement or layout of an image processing system,having an optical sensor 10, particularly of a standard camera sensitiveto the near infrared, which releases raw images to a processing unit 12.Processing unit 12, which, depending on the exemplary embodiment, ismade up of separate signal processing units or of one digital processor,has an image sharpening module 14 as well as a contrast enhancementmodule 16. Image sharpening module 14 converts the raw images deliveredby camera 10 into sharpened images, which are released to the contrastenhancement module. Contrast enhancement module 16 is made up of animaging function 18, which may be a nonlinear characteristic curve,which converts the sharpened images delivered by image sharpening module14 into high-contrast images adapted to display 20 used. In addition, animaging function calculating module 22 is a component of the contrastenhancement module. On the basis of the sharpened images delivered byimage sharpening module 14, it ascertains or determines the imagingfunction (characteristic curve 18). Contrast enhancement module 16 thenoutputs the images, weighted by characteristic curve 18 and sharpened,as contrast-enhanced images to display 20 for display.

As to camera 10, this may be a “standard” camera which is used invarious applications, and which is sensitive in the visible range and inthe near infrared. The sensitivity to the near infrared, in thiscontext, is achieved, for example, by doing without filtering measuresthat are at times provided in such standard cameras. Display 20 involvesa display arrangement which can show a lesser number of gray-scalevalues than the camera. An example is an LCD display, on which theimages recorded by the camera, especially the night vision images of themotor vehicle surroundings are shown visibly to the driver.

Depending on the particular arrangement, an image sharpening module anda contrast enhancement module may be used jointly or separately (i.e.only one of the modules is used). The sequence of the procedure, firstimage sharpening, then contrast enhancement, is expedient when bothmodules are used for image processing.

The system shown in FIG. 1, in an exemplary embodiment, represents anight vision system for motor vehicle applications in which the“standard” camera takes an image in the near infrared in combinationwith a suitable illumination of the road area in front of the vehicle.The illumination may, for instance, come from a vehicle headlight orfrom an additional floodlight having infrared light components. Thenight vision images taken are then displayed to the driver with the aidof the image processing shown. In the exemplary embodiment, a camera isused which can show up to 4096 gray-scale values, whereas the displayused can typically only show 64 gray-scale values.

In the exemplary embodiment, image sharpening module 14 works with theaid of an image sharpening method in which the edges of the image, whichappear blurred by the unsharp imaging, are enhanced. In the exemplaryembodiment, so-called inverse filtering is used as the method. In thiscontext, the unsharpness of the imaging in the near infrared isdescribed by a so-called point-spread function or by itsFourier-transformed function, the modulation transfer function. In theexemplary embodiment, this function is measured for the type of cameraby a standard method at an infrared illumination adjusted to thevehicle's headlight. This modulation transfer function is inverted andtransformed back again into local space so that a filter mask isobtained which in the ideal case exactly compensates for the unsharpnessdue to reflection.

In this context, with respect to the spatial frequencies, this filtermask has high pass characteristics, i.e. lower spatial frequencies aredamped more strongly than higher spatial frequencies. An exactcompensation of the unsharpness conditioned by reflection, as a rule,cannot be achieved in practice by quantization effects and/or saturationeffects, so that some detail is lost. It is believed, however, that thisis minor when using a display of low resolution, since the loss ofdetail is not discernable by the viewer. Therefore, the method removesthe unsharpness that may be present in the imaging in a sufficientmeasure.

Consequently, from the image sharpening module, upon photographing atest image (such, for example, as the well known Siemens Star), thereresults an enhanced contrast of the image in the range of the shortspatial wavelengths, i.e. the higher frequencies. This effect becomesclearer if, in the test sample, a sine-shaped instead of a rectangularcurve is specified, since in that case the influence of the higherharmonics drops out, and furthermore the amplitude of the intensityfluctuation is not a maximum.

For contrast enhancement, two methods for calculating an imagingfunction (characteristic curve) are shown below, with whose assistancethe gray-scale values of the (possibly sharpened) camera imaging areconverted to gray-scale values of the display. Which of the two methodsis used depends on the specific application.

A first method for contrast enhancement is shown in the light of thediagram in FIG. 2. In this diagram the gray-scale values of display GWDis plotted against the gray-scale values of camera GWK. The point ofdeparture is histogram 100 of the gray-scale values of the camera image,which is also drawn in FIG. 2. In this histogram, the frequency of theindividual gray-scale values in the analyzed image is plotted. Fromhistogram 100 there is then derived a gray-scale value range of thecamera image, which covers a predefined percentage of the gray-scalevalues of the camera image.

In the exemplary embodiment, the limits of these gray-scale value rangesare established so that a predetermined percentage of the low gray-scalevalues and a certain percentage of the highest gray-scale values of thecamera image (e.g. 5% each) do not fall within the designated segment.From these so-called “p” percentiles of the histogram, the position ofthe segment borders (minimum and maximum gray-scale value) is determinedfor each camera image. Based on the calculated segment borders, acharacteristic curve 102 is then derived, which in the variant shown ispiecewise linear, and which images m gray-scale values of the cameraimaging per n gray-scale values of the display (as a rule, m being >n).

In this context, the characteristics curve or the imaging function ismade up of three sections: in the first section, all camera gray-scalevalues are imaged at the lowest gray-scale value of the display, in themiddle section, m gray-scale values of the camera image are imaged at ngray-scale values of the display, and in the last section, all cameragray-scale values are imaged at the highest display gray-scale value.Since the gray-scale values of the camera image that are to be imagedhave a higher number than that of the display, it is therefore throughthe linear part of function 102 that gray-scale value ranges of thecamera are converted to a gray-scale value of the display. Thus thefunction executes a “clipping” operation, and compresses or spreadsapart the middle gray-scale value range of the camera imaging to thedisplay gray-scale values.

A second method is shown in the light of the illustrations in FIGS. 3 aand 3 b. This method is denoted as a histogram adjustment. In thisinstance, the cumulative distribution function is formed from histogram100, by adding on the histogram values one after another (integrating).This is shown in FIG. 3 a, in which the frequency of the cameragray-scale values is plotted against the possible camera gray-scalevalues. From this distribution function (histogram) then, according toFIG. 3 b, a nonlinear characteristics curve or imaging function shown inFIG. 3 b is ascertained by adding on the frequency values.

This characteristics curve serves as the basis of the conversion of thegray-scale values range of camera GWK to the gray-scale values GWD ofthe display. The imaging curve shown allocates to every gray-scale valuerange of the camera image the optimal resolution in the gray-scale valuerange of the display. Since the gray-scale number of the display is lessthan that of the camera image, the imaging curve has to be subsampledwith the number of gray-scale values available in the display. In thismanner, certain gray-scale value ranges of the camera have only onegray-scale value assigned to them. To more frequent camera gray-scalevalues (see area of greater rise in the characteristic curve) a greaternumber of display gray-scale values is assigned (higher resolution), andto less frequently appearing camera gray-scale values (flattercharacteristic curve area) a lesser number of display gray-scale valuesis assigned.

Depending on the exemplary embodiment, the characteristic curve or theimaging function is calculated anew according to one of the namedmethods for each image or for every n^(th) image. The imaging functionmay be low-pass filtered, so as to suppress the noise in the histogram.

The advantages of using such an adaptive, nonlinear characteristicscurve or imaging function may be demonstrated likewise when images aretaken of test samples. If, for example, two gray-scale value wedgeshaving different upward slopes of the intensity profile arephotographed, the adaptive characteristic curve or imaging functionachieves that both test samples in the image shown appear the same.

In the exemplary embodiment, the image processing (image sharpening andcontrast enhancement) is carried out or performed in one computer, whichconverts the raw images delivered by the camera into high-contrast,sharpened images for display indication. In this context, the procedureexplained above is implemented as a computer program. An example of sucha computer program is shown in the flow diagram in FIG. 4.

The raw image delivered by the camera is read in according to step 200.In step 202 the image is filtered using the specified filter mask. Thisrepresents the sharpened image. From this image, according to step 206,the histogram is derived, and in step 208 the imaging function isgenerated according to one of the methods described above. Thereafter,in step 210, the sharpened image is weighted with the imaging functionthat was ascertained, i.e. gray-scale value ranges of the image areconverted into a display gray-scale value according to the imagingfunction, and then, in step 212, are output to the display. In thiscontext, the program sketched using the flow diagram is run through foreach raw image. In one exemplary embodiment, the imaging function isascertained only for one predetermined number of images (e.g. for everytenth one).

An example application of the procedural method shown is a night visionsystem for motor vehicles. However, the procedural method shown may alsobe applied outside of this application case, for other night visionapplications, or in image processing systems in the near infrared or inthe visible range, in which unsharp images are created or images of highresolution are shown using displays having low resolution.

In addition, the procedural method described may also be applied inconnection with colored images, or in any other suitable application.

1. A night vision system for a motor vehicle, comprising: a camera thatis sensitive to near infrared radiation; an evaluation arrangement toimprove a raw image, obtained by the camera, to provide an improvedimage; and a display arrangement having a lower gray-scale valueresolution compared to that of the camera to display the improved image,wherein a target area to be imaged using the camera is illuminated byone of a vehicle headlight and a floodlight, wherein the raw imagedelivered by a sensor is first sharpened and then a contrast of thesharpened image is enhanced, and wherein the sharpening of the image isachieved by an image processing module, which sharpens the unsharp imagecreated by a reflection of an incident infrared light at a rear wall ofa housing of an optical sensor.
 2. The night vision system of claim 1,wherein the camera is a CMOS camera.
 3. The night vision system of claim1, wherein a target area to be imaged using the camera is illuminated byone of a vehicle headlight and a floodlight.
 4. The night vision systemof claim 1, wherein the raw image delivered by a sensor is firstsharpened and then a contrast of the sharpened image is enhanced.
 5. Thenight vision system of claim 4, wherein the sharpening of the image isachieved by an image processing module, which sharpens the unsharp imagecreated by a reflection of an incident infrared light at a rear wall ofa housing of an optical sensor.
 6. Then night vision system of claim 4,wherein a contrast of the sharpened image is enhanced by an adaptiveimaging function which is a nonlinear characteristic function.
 7. Thenight vision system of claim 1, wherein an image sharpening module and acontrast enhancement module may be used jointly or separately.
 8. Thenight vision system of claim 1, wherein an unsharpness of the raw imagein the near infrared is described by a point-spread function or by itsFourier-transform function.
 9. The night vision system of claim 1,wherein a characteristics curve or an imaging function is made up ofthree sections, and wherein in the first section all camera gray-scalevalues are imaged at a lowest gray-scale value of the display.
 10. Thenight vision system of claim 1, wherein an image sharpening module and acontrast enhancement module may be used jointly or separately, andwherein an unsharpness of the raw image in the near infrared isdescribed by a point-spread function or by its Fourier-transformfunction, wherein a characteristics curve or an imaging function is madeup of three sections, and wherein in the first section all cameragray-scale values are imaged at a lowest gray-scale value of thedisplay.
 11. The night vision system of claim 1, wherein a target areato be imaged using the camera is illuminated by one of a vehicleheadlight and a floodlight, wherein the raw image delivered by a sensoris first sharpened and then a contrast of the sharpened image isenhanced, and wherein a contrast of the sharpened image is enhanced byan adaptive imaging function which is a nonlinear characteristicfunction.
 12. The night vision system of claim 11, wherein an imagesharpening module and a contrast enhancement module may be used jointlyor separately, and wherein an unsharpness of the raw image in the nearinfrared is described by a point-spread function or by itsFourier-transform function, wherein a characteristics curve or animaging function is made up of three sections, and wherein in the firstsection all camera gray-scale values are imaged at a lowest gray-scalevalue of the display.